{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4f8e4282-defd-4657-9bc8-92e316184158",
   "metadata": {},
   "source": [
    "# MICRO 2023 TeAAL Artifact Evaluation Figures 8 and 9\n",
    "\n",
    "## TeAAL: A Declarative Framework for Modeling Sparse Tensor Accelerators\n",
    "\n",
    "This notebook calls the scripts necessary to simulate four state-of-the-art sparse tensor algebra accelerators: ExTensor [15], Gamma [52], OuterSPACE [36], and SIGMA [33], and generate the graphs found in Figures 9, 10, and 11 of the [TeAAL paper](https://arxiv.org/abs/2304.07931).\n",
    "\n",
    "**Warning: Running a cell more than once will overwrite the previous output produced by the cell. Please back up all results as soon as they are produced in case you accidentally rerun the cell.** All results are stored in `data/generated/`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14722cf2-f92c-4106-89a8-cdc3cd3959a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import os\n",
    "import sys\n",
    "sys.path.insert(0, \"/home/workspace\")\n",
    "\n",
    "import scripts.extensor as extensor\n",
    "import scripts.extensor_energy as extensor_energy\n",
    "import scripts.gamma as gamma\n",
    "import scripts.outerspace as outerspace\n",
    "import scripts.sigma as sigma"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0a81d7e-2d27-4131-a8c4-a8adf351d887",
   "metadata": {},
   "source": [
    "Set `pregenerated` to `True` to use pre-generated results for the figure and to `False` to display results you generated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b050c1ce-fbdc-44df-930f-77d3d2c4b43d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipywidgets import interactive\n",
    "\n",
    "pregenerated = False\n",
    "\n",
    "def set_pregenerated(**kwargs):\n",
    "    global pregenerated\n",
    "    pregenerated = kwargs['pregenerated']\n",
    "\n",
    "options = {\"pregenerated\": [False, True]}\n",
    "\n",
    "w = interactive(set_pregenerated, **options)\n",
    "\n",
    "display(w)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f65291e-699d-4eab-bfbe-cf74bb742b78",
   "metadata": {},
   "source": [
    "## Figure 9A\n",
    "\n",
    "### Simulate ExTensor with TeAAL\n",
    "\n",
    "Expected simulation time: 28 hours"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb142ad9-b70c-4c3c-8292-d30b55ab6745",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%python3 -O\n",
    "\n",
    "import os\n",
    "import sys\n",
    "sys.path.insert(0, \"/home/workspace\")\n",
    "\n",
    "import scripts.extensor as extensor\n",
    "extensor.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a9c4361-9848-4c27-9817-97da227394c2",
   "metadata": {},
   "source": [
    "### Display Figure 9A\n",
    "\n",
    "To display Figure 9A with pre-generated data (i.e. without running the TeAAL simulation), toggle the `pregenerated` widget above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce82df8f-1093-4db0-bd8b-8db557997642",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = extensor.graph_mem(pregenerated=pregenerated)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c5b1a46-be3b-464f-a98e-4d7155eb33ef",
   "metadata": {},
   "source": [
    "## Figure 9B\n",
    "\n",
    "### Simulate Gamma with TeAAL\n",
    "\n",
    "Expected simulation time: 10 hours"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5700441-614d-487f-a16e-424dc9a66f4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%python3 -O \n",
    "\n",
    "import os\n",
    "import sys\n",
    "sys.path.insert(0, \"/home/workspace\")\n",
    "\n",
    "import scripts.gamma as gamma\n",
    "gamma.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8531e353-9a32-4838-a94e-6a76c5b5f753",
   "metadata": {},
   "source": [
    "### Display Figure 9B\n",
    "\n",
    "To display Figure 9B with pre-generated data (i.e. without running the TeAAL simulation), toggle the `pregenerated` widget above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90bcd958-11c7-4171-93eb-7fb1ad1886ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = gamma.graph_mem(pregenerated=pregenerated)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f29551e3-77fa-41b9-ba7a-45e5abde3f19",
   "metadata": {},
   "source": [
    "## Figure 9C\n",
    "\n",
    "### Simulate OuterSPACE with TeAAL\n",
    "\n",
    "Expected simulation time: 5 hours"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcf214ae-9a4e-4826-87d0-826401524969",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%python3 -O \n",
    "\n",
    "import os\n",
    "import sys\n",
    "sys.path.insert(0, \"/home/workspace\")\n",
    "\n",
    "import scripts.outerspace as outerspace\n",
    "outerspace.eval_mem()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6dd74895-6a1c-429c-b09a-b036f76f32ac",
   "metadata": {},
   "source": [
    "### Display Figure 9C\n",
    "\n",
    "To display Figure 9C with pre-generated data (i.e. without running the TeAAL simulation), toggle the `pregenerated` widget above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f96fd3a-fbdb-4622-8fc5-e35be94dff72",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = outerspace.graph_mem(pregenerated=pregenerated)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b6d19dd-00dd-481f-a67f-7cdd7d969242",
   "metadata": {},
   "source": [
    "## Figure 10A\n",
    "\n",
    "The TeAAL experiments for Figure 10A should have already been run as a part of Figure 9A.\n",
    "\n",
    "### Simulate ExTensor with TeAAL\n",
    "\n",
    "Uncomment and run the below cell only if you have not evaluated Figure 9A (see above).\n",
    "\n",
    "Expected simulation time: 28 hours"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3238ad13-1353-4210-9723-e7035a2438ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "# %%python3 -O \n",
    "# import os\n",
    "# import sys\n",
    "# sys.path.insert(0, \"/home/workspace\")\n",
    "\n",
    "# import scripts.extensor as extensor\n",
    "# extensor.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54624b00-1aa5-4a86-9617-41075ebf0c59",
   "metadata": {},
   "source": [
    "### Simulate ExTensor with Sparseloop\n",
    "\n",
    "Expected simulation time: 1 minute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "215c5682-84b6-4b4d-a7b6-9a208b25f9f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "extensor.run_sparseloop()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b4066ee-ae36-4865-acd7-2d1e510bd136",
   "metadata": {},
   "source": [
    "### Display Figure 10A\n",
    "\n",
    "To display Figure 10A with pre-generated data (i.e. without running the TeAAL simulation), toggle the `pregenerated` widget above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3969dc9-edab-4228-8ef0-3dbce35923c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = extensor.graph_speedup(pregenerated=pregenerated)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fccd1977-5f7f-4ff9-95c6-fe149c3b8c61",
   "metadata": {},
   "source": [
    "## Figure 10B\n",
    "\n",
    "### Simulate Gamma with TeAAL\n",
    "\n",
    "Uncomment and run the below cell only if you have not evaluated Figure 9B (see above).\n",
    "\n",
    "Expected simulation time: 10 hours"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b9a4cde-889f-4a09-94ca-b51f935f02a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# %%python3 -O \n",
    "\n",
    "# import os\n",
    "# import sys\n",
    "# sys.path.insert(0, \"/home/workspace\")\n",
    "\n",
    "# import scripts.gamma as gamma\n",
    "# gamma.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e10a1ae-d497-41a3-88aa-85aea97adef6",
   "metadata": {},
   "source": [
    "### Display Figure 10B\n",
    "\n",
    "To display Figure 10B with pre-generated data (i.e. without running the TeAAL simulation), toggle the `pregenerated` widget above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5087f6aa-61d1-40bb-92ae-44f9448efe58",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = gamma.graph_speedup(pregenerated=pregenerated)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48ca7e39-a3a1-4699-8fb8-c3b86d22bc3d",
   "metadata": {},
   "source": [
    "## Figure 10C\n",
    "\n",
    "### Simulate OuterSPACE with TeAAL\n",
    "\n",
    "This simulation runs on different data than Figure 9C. Please run the below cell. \n",
    "\n",
    "Because this experiment evaluates randomly generated data, it will produce slightly different results every time. The results in the paper come from taking the median over 10 iterations.\n",
    "\n",
    "Expected simulation time: 2 hours"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74bb7648-44e9-493f-b226-d34342051548",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%python3 -O \n",
    "\n",
    "import os\n",
    "import sys\n",
    "sys.path.insert(0, \"/home/workspace\")\n",
    "\n",
    "import scripts.outerspace as outerspace\n",
    "outerspace.eval_time()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c69f70fa-9cf0-4739-a7d2-6518d0d7ee60",
   "metadata": {},
   "source": [
    "### Display Figure 10C\n",
    "\n",
    "To display Figure 10C with pre-generated data (i.e. without running the TeAAL simulation), toggle the `pregenerated` widget above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1875b520-b10f-4ff6-9dda-9a4acc002203",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = outerspace.graph_time(pregenerated=pregenerated)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "003c5c4f-ab5e-4714-aed2-c29896d9a8c2",
   "metadata": {},
   "source": [
    "## Figure 10D\n",
    "\n",
    "### Simulate SIGMA with TeAAL\n",
    "\n",
    "Because this experiment evaluates randomly generated data, it will produce slightly different results every time. The results in the paper come from taking the median over 10 iterations.\n",
    "\n",
    "Expected simulation time: 23 hours"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1980125d-7a7e-4836-bdbe-99b64e99f294",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%python3 -O \n",
    "\n",
    "import os\n",
    "import sys\n",
    "sys.path.insert(0, \"/home/workspace\")\n",
    "\n",
    "import scripts.sigma as sigma\n",
    "sigma.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a5f294c-e0c3-48c1-811f-ecc3e227cdbc",
   "metadata": {},
   "source": [
    "### Display Figure 9D\n",
    "\n",
    "To display Figure 9D with pre-generated data (i.e. without running the TeAAL simulation), toggle the `pregenerated` widget above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa49e9e8-193a-4229-ac75-a619db0a0e62",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = sigma.graph_speedup(pregenerated=pregenerated)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d431a872-ff8b-4b49-b190-b447fc7abb4f",
   "metadata": {},
   "source": [
    "## Figure 11\n",
    "\n",
    "### Simulate ExTensor with TeAAL\n",
    "\n",
    "Expected simulation time: 49 hours\n",
    "\n",
    "The datasets simulated are:\n",
    "- `wiki-Vote`\n",
    "- `p2p-Gnutella31`\n",
    "- `ca-CondMat`\n",
    "- `poisson3Da`\n",
    "- `email-Enron`\n",
    "\n",
    "Pass the name of the desired dataset as a parameter to `extensor_energy.eval()` as a string (e.g., `dataset=\"wiki-Vote\"`) or `\"all\"` for all datasets.\n",
    "\n",
    "Note: the ExTensor model for this figure is more detailed than that for Figures 9A and 10A to improve the simulation time. However, the memory traffic and execution times produced by this model and the one used above are the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c8f8bb4-ba6b-4dde-a1b2-9666956335c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%python3 -O\n",
    "\n",
    "import os\n",
    "import sys\n",
    "sys.path.insert(0, \"/home/workspace\")\n",
    "\n",
    "import scripts.extensor_energy as extensor_energy\n",
    "extensor_energy.eval(dataset=\"all\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "510044db-e914-4c62-8c01-18fea946d018",
   "metadata": {},
   "source": [
    "### Display Figure 11\n",
    "\n",
    "To display Figure 11 with pre-generated data (i.e. without running the TeAAL simulation), toggle the `pregenerated` widget above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3626434f-f10a-426b-ac4b-1016789bdd2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = extensor_energy.graph_energy(pregenerated=pregenerated)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2af1695c-80da-4f3d-b4d3-e1128db4bfdf",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
